Add tool to generate services.kt file that contains mmv1 and tpgtools/DCL services by SarahFrench · Pull Request #8344 · GoogleCloudPlatform/magic-modules

SarahFrench · 2023-07-13T21:12:54Z

Problem

Currently this PR #8254 generates the .teamcity folder in the downstream TPG and TPGB repos, and the generated code creates projects and build configurations in TeamCity without issue. One of the files needed for this is a map of the service packages, found in .teamcity/components/generated/services.kt

However, when this map of services is generated it contains only services generated by MMv1 and lacks services generated by tpgtools/DCL. This be because MMv1 generates files before tpgtools (see ordering here), so when the files in the .teamcity/ folder are generated the DCL services aren't created yet. The generator code iterates through folders in ./google/services/ and finds an incomplete list.

Proposed solution

The solutions for this could be:

Make the file mostly manually curated, with unless version == 'ga' logic controlling inclusion of beta-only services
- This is not a good idea as it will fall out of date
Move generation of the services.kt file to tpgtools/DCL as it runs last in the generation process, so it will contain all services
Update tpgtools/DCL to generate a second map of services, e.g. services-dcl.kt file, and update the Kotlin code to merge the two maps together when it creates build configs for each service.
Create a separate tool to create the single services.kt file after both MMv1 and tpgtools/DCL are finished

I've chosen to do the last option of a separate tool, because this can be deleted away from the repo once all the DCL services are migrated to MMv1, all without any changes to the existing Kotlin code.

If this PR is for Terraform, I acknowledge that I have:

Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
Generated Terraform providers, and ran make test and make lint in the generated providers to ensure it passes unit and linter tests.
Ran relevant acceptance tests using my own Google Cloud project and credentials (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
Read Write release notes before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

…/DCL services

modular-magician · 2023-07-13T21:27:00Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

SarahFrench · 2023-07-13T21:31:23Z

👆 not sure if there are no diffs due to an error in my work, or protection in the CI against arbitrary code being executed

modular-magician · 2023-07-13T22:11:35Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

zli82016 · 2023-07-13T23:47:43Z

👆 not sure if there are no diffs due to an error in my work, or protection in the CI against arbitrary code being executed

I tested it locally and don't see errors. The DCL service packages are generated in the downstream providers.

zli82016

LGTM. Thanks so much for the quick fix.

modular-magician · 2023-07-13T23:52:35Z

Tests analytics

Total tests: 2853
Passed tests 2553
Skipped tests: 297
Affected tests: 3

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccComputeFirewallPolicyRule_multipleRules|TestAccSpannerDatabaseIamPolicy|TestAccContainerAwsNodePool_BetaBasicHandWritten

Get to know how VCR tests work

modular-magician · 2023-07-13T23:56:21Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccSpannerDatabaseIamPolicy[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

SarahFrench · 2023-07-14T13:47:07Z

@zli82016 I'm a bit nervous about merging this - is it definitely not going to impact the alpha/google-private provider?

Edit: 😅 I realised that the make terraform command isn't actually used in the scripts to generate the downstream, and instead MMv1 is generated using bundle and tpgtools is triggered via its Make target. My commit below is trying to add use of make teamcity-servicemap-generate in a way that only affects TPG and TPGB

zli82016 · 2023-07-14T16:33:15Z

@zli82016 I'm a bit nervous about merging this - is it definitely not going to impact the alpha/google-private provider?

Edit: 😅 I realised that the make terraform command isn't actually used in the scripts to generate the downstream, and instead MMv1 is generated using bundle and tpgtools is triggered via its Make target. My commit below is trying to add use of make teamcity-servicemap-generate in a way that only affects TPG and TPGB

Yeah, I think it does not affect impact alpha provider.

modular-magician · 2023-07-14T18:26:12Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 1 file changed, 9 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 1 file changed, 8 insertions(+), 1 deletion(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

modular-magician · 2023-07-14T20:07:51Z

Tests analytics

Total tests: 2854
Passed tests 2555
Skipped tests: 297
Affected tests: 2

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccContainerAwsNodePool_BetaBasicHandWritten|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-07-14T20:10:46Z

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

modular-magician · 2023-07-14T20:41:27Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

modular-magician · 2023-07-14T22:04:01Z

Tests analytics

Total tests: 2856
Passed tests 2555
Skipped tests: 299
Affected tests: 2

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccContainerAwsNodePool_BetaBasicHandWritten|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-07-14T22:06:40Z

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

…/DCL services (#8344) * Add tool to generate services.kt file that contains mmv1 and tpgtools/DCL services * Fix log function call * Correct indentation size in service map output * Call teamcity Make target when generating TPG/TPGB * Add commas to generated map string * test change * update test * remove test file --------- Co-authored-by: Shuya Ma <shuyama@google.com>

…/DCL services (GoogleCloudPlatform#8344) * Add tool to generate services.kt file that contains mmv1 and tpgtools/DCL services * Fix log function call * Correct indentation size in service map output * Call teamcity Make target when generating TPG/TPGB * Add commas to generated map string * test change * update test * remove test file --------- Co-authored-by: Shuya Ma <shuyama@google.com>

Add tool to generate services.kt file that contains mmv1 and tpgtools…

56f6e2b

…/DCL services

SarahFrench requested a review from zli82016 July 13, 2023 21:13

SarahFrench commented Jul 13, 2023

View reviewed changes

Comment thread tools/teamcity-generator/main.go

SarahFrench and others added 2 commits July 13, 2023 22:52

Fix log function call

9ebc7a1

Correct indentation size in service map output

1ead2a5

zli82016 approved these changes Jul 13, 2023

View reviewed changes

Call teamcity Make target when generating TPG/TPGB

ddc1d43

SarahFrench requested a review from shuyama1 July 14, 2023 14:09

SarahFrench commented Jul 14, 2023

View reviewed changes

Comment thread .ci/containers/downstream-builder/generate_downstream.sh

Add commas to generated map string

90a64d8

shuyama1 added 2 commits July 14, 2023 10:41

test change

2d79141

update test

fcc4a37

remove test file

a42b787

shuyama1 approved these changes Jul 19, 2023

View reviewed changes

SarahFrench merged commit d637a19 into main Jul 19, 2023

SarahFrench deleted the add-teamcity-servicemap-generator branch March 19, 2024 21:59

Uh oh!

Conversation

SarahFrench commented Jul 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Proposed solution

Uh oh!

modular-magician commented Jul 13, 2023

Diff report

Uh oh!

SarahFrench commented Jul 13, 2023

Uh oh!

Uh oh!

modular-magician commented Jul 13, 2023

Diff report

Uh oh!

zli82016 commented Jul 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zli82016 left a comment

Choose a reason for hiding this comment

Uh oh!

modular-magician commented Jul 13, 2023

Tests analytics

Action taken

Uh oh!

modular-magician commented Jul 13, 2023

Rerun these tests in REPLAYING mode to catch issues

Uh oh!

SarahFrench commented Jul 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

zli82016 commented Jul 14, 2023

Uh oh!

modular-magician commented Jul 14, 2023

Diff report

Uh oh!

modular-magician commented Jul 14, 2023

Tests analytics

Action taken

Uh oh!

modular-magician commented Jul 14, 2023

Uh oh!

modular-magician commented Jul 14, 2023

Diff report

Uh oh!

modular-magician commented Jul 14, 2023

Tests analytics

Action taken

Uh oh!

modular-magician commented Jul 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

SarahFrench commented Jul 13, 2023 •

edited

Loading

zli82016 commented Jul 13, 2023 •

edited

Loading

SarahFrench commented Jul 14, 2023 •

edited

Loading